Skip to content

[DRAFT] fix(ci): resolve pre-existing CI failures on rhoai-3.3#2184

Open
crackcodecamp wants to merge 5 commits intored-hat-data-services:rhoai-3.3from
crackcodecamp:fix/ci-preexisting-failures-rhoai-3.3
Open

[DRAFT] fix(ci): resolve pre-existing CI failures on rhoai-3.3#2184
crackcodecamp wants to merge 5 commits intored-hat-data-services:rhoai-3.3from
crackcodecamp:fix/ci-preexisting-failures-rhoai-3.3

Conversation

@crackcodecamp
Copy link
Copy Markdown

@crackcodecamp crackcodecamp commented Apr 30, 2026

Summary

Fixes pre-existing CI failures affecting all PRs targeting rhoai-3.3.

Root cause

Since November 2025, GitHub Actions runs pull_request_target workflows from main against every branch. This means CI checks added to main after rhoai-3.3 branched off are now running against it — referencing code, actions, and configs the branch doesn't have.

Fixes

Issue Commits Details
v3-0 Tekton pipeline build-args (17 subtests) fc5985c v3-0 pipelines referenced non-konflux. build-args while using Dockerfile.konflux.* — updated all 17 to use konflux.* build-args
Stale wheel exception e0c5f5f All pyproject.toml files now consistently use wheel~=0.46.2, so the ignored_exceptions entry is no longer needed
Pillow CVE vs llmcompressor cap 6f2ea38 llmcompressor==0.9.0 caps pillow<=12.0.0 but CVE-2026-40192 requires >=12.2.0 — added override-dependencies and regenerated lock files
Missing composite actions a27390d build-notebooks-TEMPLATE.yaml on main references apt-install, setup-uv, playwright-test actions that were missing from rhoai-3.3 — backported all three
provision-k8s runner failure (24 AIPCC builds) 9bf5093 sudo rm /etc/apt/sources.list.d/microsoft-prod.list (without -f) fails on Ubuntu 24.04 runners where the file no longer exists. Switched to use the backported apt-install action which uses rm -f
pytest-tests pre-commit failure 9bf5093 uv-pre-commit pinned at rev 0.9.18 conflicts with required-version >= 0.10.9 in pyproject.toml. Switched to a local hook using system uv (installed by setup-uv)

Remaining (not fixable here)

  • Snyk checks — external service, unrelated to workflow mismatch
  • Tide — merge conflict with base, will resolve after rebase/merge

Test plan

  • pytest-tests passes (all subtests green, pre-commit uv-lock succeeds)
  • check-generated-code passes (llmcompressor lock generation succeeds)
  • AIPCC image builds pass (provision-k8s no longer fails on microsoft-prod.list)
  • code-static-analysis passes

All v3-0 Tekton PipelineRun YAMLs that reference Dockerfile.konflux.*
were incorrectly pointing to non-konflux build-args files (cpu.conf,
cuda.conf, rocm.conf). The test_rhds_pipelines_use_rhds_args test
enforces that Dockerfile.konflux.* must use konflux.* build-args.

Update all 17 affected v3-0 pipelines to reference the konflux.*
prefixed build-args files which already exist in each directory.
All pyproject.toml files now use wheel~=0.46.2 consistently (the
==0.46.2 variant was removed). The ignored_exceptions entry for wheel
is stale and causes test_image_pyprojects_version_alignment to fail
with "now has single specifier but ignored_exceptions expects multiple".
@openshift-ci openshift-ci Bot requested review from atheo89 and jiridanek April 30, 2026 09:10
@openshift-ci
Copy link
Copy Markdown

openshift-ci Bot commented Apr 30, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign jiridanek for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 30, 2026

📝 Walkthrough

Walkthrough

Adds three new composite GitHub Actions (apt-install, playwright-test, setup-uv); switches many Tekton PipelineRun manifests to Konflux-specific build-args files; tweaks Dockerfiles (Meson build-constraint handling, gettext, PDF deps script); updates uv and Pillow settings in pyproject.toml files; removes wheel from a test's ignored exceptions; updates pre-commit hook and a provisioning workflow step.

Changes

GitHub Actions – New Composite Actions

Layer / File(s) Summary
Action declarations
.github/actions/apt-install/action.yml, .github/actions/playwright-test/action.yml, .github/actions/setup-uv/action.yml
Adds three composite actions: apt-install (inputs: packages, update; configures APT/dpkg, optionally runs apt-get update, installs packages), playwright-test (inputs/outputs for running Playwright tests in a container via Podman, optional report upload, exposes outcome), setup-uv (reads .python-version, emits version, invokes astral-sh/setup-uv with caching).

Tekton PipelineRun Build-Args Refactoring

Layer / File(s) Summary
Parameter updates
.tekton/odh-pipeline-runtime-*-push.yaml, .tekton/odh-workbench-jupyter-*-push.yaml
Updates spec.params[name="build-args-file"].value across many PipelineRun manifests to use konflux.cpu.conf, konflux.cuda.conf, or konflux.rocm.conf instead of the previous generic cpu.conf/cuda.conf/rocm.conf.

Dockerfile Constraint and Dependency Updates

Layer / File(s) Summary
Build constraint wiring
codeserver/ubi9-python-3.12/Dockerfile.cpu
Appends meson<1.11 to constraint.txt and passes it via --build-constraint to uv pip install in the whl-cache stage (replaces previous overwrite flow).
Runtime deps and PDF deps script
codeserver/.../Dockerfile.cpu, jupyter/datascience/ubi9-python-3.12/Dockerfile.cpu, .../Dockerfile.konflux.cpu
Adds gettext to final codeserver stage packages; replaces install_with_retry.sh texlive-install with utils/install_pdf_deps.sh for PDF export deps in datascience images.

Python Configuration and Dependency Overrides

Layer / File(s) Summary
Root uv requirement
pyproject.toml
Adds [tool.uv].required-version = ">=0.10.9".
Pillow override
jupyter/pytorch+llmcompressor/.../pyproject.toml, runtimes/pytorch+llmcompressor/.../pyproject.toml
Adds tool.uv.override-dependencies = ["pillow>=12.2.0"] for pytorch+llmcompressor variants.

Tests / Tooling / CI wiring

Layer / File(s) Summary
Tests
tests/test_main.py
Removes wheel from ignored_exceptions in test_image_pyprojects_version_alignment, so wheel must now resolve to a single specifier across pyprojects.
Workflows and provisioning wiring
.github/actions/provision-k8s/action.yml, .github/actions/apt-install/action.yml
Provisioning workflow: adds an "Install prerequisites" step that calls the new apt-install action; replaces a previously extensive "Install cri-o" step body with a minimal stub (strict Bash options only).
Pre-commit hook
.pre-commit-config.yaml
Replaces remote uv pre-commit hook with a local uv-lock hook using system uv (uv lock --locked), targeting uv.lock, pyproject.toml, uv.toml, and disabling pass_filenames.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Suggested reviewers

  • dibryant
  • jiridanek
  • daniellutz
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description lacks required sections: 'How Has This Been Tested?' is empty and the self-checklist items are not checked. Complete the 'How Has This Been Tested?' section with detailed testing steps and check off the self-checklist items to confirm testing was performed.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: fixing pre-existing CI failures on rhoai-3.3, which aligns with the substantial changeset addressing Tekton pipelines, test exceptions, dependencies, and backported actions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Copy Markdown

@crackcodecamp — This PR is from a fork.
The build-rhoai CI job was skipped because subscription
builds (RHEL, AIPCC) need secrets unavailable to forks.
ODH builds and code quality checks still ran.

Recommended: Push your branch to the main repo for full CI:

git remote add upstream https://github.com/red-hat-data-services/notebooks.git
git push upstream HEAD:crackcodecamp/your-branch-name

Then open a new PR from that branch.

No push access? A maintainer will cherry-pick and test your changes.

See CONTRIBUTING.md for details.

Add pillow>=12.2.0 to override-dependencies in both llmcompressor
pyproject.toml files to resolve the dependency conflict between
llmcompressor's pillow upper bound and the required pillow version.

Regenerate all pylock.toml files with latest uv to fix stale lock
files that included non-Linux platform wheels.

Sync codeserver Dockerfile.cpu with Dockerfile.konflux.cpu to fix
the Dockerfile alignment check (meson constraint file naming and
missing gettext package).

Regenerate Dockerfile fragments via dockerfile_fragments.py.
@crackcodecamp crackcodecamp force-pushed the fix/ci-preexisting-failures-rhoai-3.3 branch from 5f21dba to d3b40c1 Compare April 30, 2026 09:37
Copy apt-install, setup-uv, and playwright-test composite actions
from upstream/main. These are referenced by the build TEMPLATE
workflow on main, which resolves at runtime. Without them, builds
fail at the 'Setup uv and Python' step.

Add required-version to root pyproject.toml so the setup-uv action
can determine which uv version to install.
@crackcodecamp crackcodecamp force-pushed the fix/ci-preexisting-failures-rhoai-3.3 branch from d3b40c1 to a27390d Compare April 30, 2026 09:46
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/actions/apt-install/action.yml:
- Line 56: The run step currently expands inputs.packages directly (run: sudo
apt-get install ${{ inputs.packages }}), allowing shell/option injection; change
the action to treat packages as a validated, tokenized list and pass them as
explicit argv rather than a raw shell expansion: require callers to supply a
static space-separated list (or change the input to a multiline/array format),
add shell safety (set -euo pipefail) in the script, and invoke apt-get with a
safe invocation that uses -- to end options and passes each package as a
separate argument (e.g., populate "$@" from a validated split of inputs.packages
or use the action.yml args array feature) so no untrusted characters are
interpreted by the shell or as apt options.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b40b4aaf-fdb5-414c-9017-85eecbe8cc47

📥 Commits

Reviewing files that changed from the base of the PR and between 1af2781 and a27390d.

📒 Files selected for processing (43)
  • .github/actions/apt-install/action.yml
  • .github/actions/playwright-test/action.yml
  • .github/actions/setup-uv/action.yml
  • .tekton/odh-pipeline-runtime-datascience-cpu-py312-v3-0-push.yaml
  • .tekton/odh-pipeline-runtime-minimal-cpu-py312-v3-0-push.yaml
  • .tekton/odh-pipeline-runtime-pytorch-cuda-py312-v3-0-push.yaml
  • .tekton/odh-pipeline-runtime-pytorch-llmcompressor-cuda-py312-v3-0-push.yaml
  • .tekton/odh-pipeline-runtime-pytorch-rocm-py312-v3-0-push.yaml
  • .tekton/odh-pipeline-runtime-tensorflow-cuda-py312-v3-0-push.yaml
  • .tekton/odh-pipeline-runtime-tensorflow-rocm-py312-v3-0-push.yaml
  • .tekton/odh-workbench-jupyter-datascience-cpu-py312-v3-0-push.yaml
  • .tekton/odh-workbench-jupyter-minimal-cpu-py312-v3-0-push.yaml
  • .tekton/odh-workbench-jupyter-minimal-cuda-py312-v3-0-push.yaml
  • .tekton/odh-workbench-jupyter-minimal-rocm-py312-v3-0-push.yaml
  • .tekton/odh-workbench-jupyter-pytorch-cuda-py312-v3-0-push.yaml
  • .tekton/odh-workbench-jupyter-pytorch-llmcompressor-cuda-py312-v3-0-push.yaml
  • .tekton/odh-workbench-jupyter-pytorch-rocm-py312-v3-0-push.yaml
  • .tekton/odh-workbench-jupyter-tensorflow-cuda-py312-v3-0-push.yaml
  • .tekton/odh-workbench-jupyter-tensorflow-rocm-py312-v3-0-push.yaml
  • .tekton/odh-workbench-jupyter-trustyai-cpu-py312-v3-0-push.yaml
  • codeserver/ubi9-python-3.12/Dockerfile.cpu
  • codeserver/ubi9-python-3.12/pylock.toml
  • jupyter/datascience/ubi9-python-3.12/Dockerfile.cpu
  • jupyter/datascience/ubi9-python-3.12/Dockerfile.konflux.cpu
  • jupyter/datascience/ubi9-python-3.12/pylock.toml
  • jupyter/minimal/ubi9-python-3.12/pylock.toml
  • jupyter/pytorch+llmcompressor/ubi9-python-3.12/pylock.toml
  • jupyter/pytorch+llmcompressor/ubi9-python-3.12/pyproject.toml
  • jupyter/pytorch/ubi9-python-3.12/pylock.toml
  • jupyter/rocm/pytorch/ubi9-python-3.12/pylock.toml
  • jupyter/rocm/tensorflow/ubi9-python-3.12/pylock.toml
  • jupyter/tensorflow/ubi9-python-3.12/pylock.toml
  • jupyter/trustyai/ubi9-python-3.12/pylock.toml
  • pyproject.toml
  • runtimes/datascience/ubi9-python-3.12/pylock.toml
  • runtimes/minimal/ubi9-python-3.12/pylock.toml
  • runtimes/pytorch+llmcompressor/ubi9-python-3.12/pylock.toml
  • runtimes/pytorch+llmcompressor/ubi9-python-3.12/pyproject.toml
  • runtimes/pytorch/ubi9-python-3.12/pylock.toml
  • runtimes/rocm-pytorch/ubi9-python-3.12/pylock.toml
  • runtimes/rocm-tensorflow/ubi9-python-3.12/pylock.toml
  • runtimes/tensorflow/ubi9-python-3.12/pylock.toml
  • tests/test_main.py
💤 Files with no reviewable changes (1)
  • tests/test_main.py


- name: Install packages
shell: bash
run: sudo apt-get install ${{ inputs.packages }}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Harden package input handling to prevent command/option injection.

Line 56 executes inputs.packages directly in the shell. In a reusable action, this can be abused via shell metacharacters or apt flags if any caller passes non-literal input.

🔧 Proposed hardening
-    - name: Install packages
-      shell: bash
-      run: sudo apt-get install ${{ inputs.packages }}
+    - name: Install packages
+      shell: bash
+      env:
+        INPUT_PACKAGES: ${{ inputs.packages }}
+      run: |
+        read -r -a pkgs <<< "${INPUT_PACKAGES}"
+        sudo apt-get install -- "${pkgs[@]}"
#!/bin/bash
set -euo pipefail

# Verify all callers of this action and inspect how `packages` is provided.
rg -n --glob '.github/workflows/*.y*ml' 'uses:\s*\./\.github/actions/apt-install' -C3
rg -n --glob '.github/workflows/*.y*ml' 'packages:\s*\$\{\{'

Expected result: callers should pass static package lists, not values derived from untrusted event payloads.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/actions/apt-install/action.yml at line 56, The run step currently
expands inputs.packages directly (run: sudo apt-get install ${{ inputs.packages
}}), allowing shell/option injection; change the action to treat packages as a
validated, tokenized list and pass them as explicit argv rather than a raw shell
expansion: require callers to supply a static space-separated list (or change
the input to a multiline/array format), add shell safety (set -euo pipefail) in
the script, and invoke apt-get with a safe invocation that uses -- to end
options and passes each package as a separate argument (e.g., populate "$@" from
a validated split of inputs.packages or use the action.yml args array feature)
so no untrusted characters are interpreted by the shell or as apt options.

@openshift-ci
Copy link
Copy Markdown

openshift-ci Bot commented Apr 30, 2026

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@crackcodecamp crackcodecamp changed the title fix(ci): resolve pre-existing CI failures on rhoai-3.3 [DRAFT] fix(ci): resolve pre-existing CI failures on rhoai-3.3 May 4, 2026
Since Nov 2025, GitHub Actions runs pull_request_target workflows from
main against all branches. This causes rhoai-3.3 CI failures because
newer workflows reference code/configs the branch doesn't have.

- provision-k8s: replace inline `sudo rm` (fails on Ubuntu 24.04 where
  microsoft-prod.list no longer exists) with the apt-install composite
  action that uses `rm -f` (idempotent)
- pre-commit: switch uv-lock from uv-pre-commit repo (pinned at 0.9.18)
  to a local hook using system uv, avoiding version mismatch with
  required-version >= 0.10.9 in pyproject.toml
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.pre-commit-config.yaml:
- Around line 11-14: The pre-commit hook definition for entry "uv lock --locked"
currently uses a root-anchored files pattern and pass_filenames: false, so it
only validates the repo root; change the hook so it runs per changed file and
matches nested projects by (1) setting pass_filenames: true and (2) replacing
the files regex that references '^uv\.lock$|^pyproject\.toml$|^uv\.toml$' with a
non-root-anchored pattern that matches nested paths (e.g. allow files ending
with uv.lock, pyproject.toml or uv.toml anywhere in the tree), keeping the entry
"uv lock --locked" unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 40a6e47e-d459-4bc3-8491-21b1a594c72f

📥 Commits

Reviewing files that changed from the base of the PR and between a27390d and 9bf5093.

📒 Files selected for processing (2)
  • .github/actions/provision-k8s/action.yml
  • .pre-commit-config.yaml

Comment thread .pre-commit-config.yaml
Comment on lines +11 to +14
entry: uv lock --locked
language: system
files: '(^uv\.lock$|^pyproject\.toml$|^uv\.toml$)'
pass_filenames: false
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

uv-lock hook currently protects only the repo root project

The files pattern is root-anchored, and with pass_filenames: false + uv lock --locked, this validates only the root lock context. Nested runtime pyproject.toml/uv.lock changes can bypass this guard.

Suggested adjustment
       - id: uv-lock
         name: uv-lock
-        entry: uv lock --locked
+        entry: bash -c 'set -euo pipefail; for f in "$@"; do d="$(dirname "$f")"; (cd "$d" && uv lock --locked); done' --
         language: system
-        files: '(^uv\.lock$|^pyproject\.toml$|^uv\.toml$)'
-        pass_filenames: false
+        files: '(^|.*/)(uv\.lock|pyproject\.toml|uv\.toml)$'
+        pass_filenames: true
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
entry: uv lock --locked
language: system
files: '(^uv\.lock$|^pyproject\.toml$|^uv\.toml$)'
pass_filenames: false
entry: bash -c 'set -euo pipefail; for f in "$@"; do d="$(dirname "$f")"; (cd "$d" && uv lock --locked); done' --
language: system
files: '(^|.*/)(uv\.lock|pyproject\.toml|uv\.toml)$'
pass_filenames: true
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.pre-commit-config.yaml around lines 11 - 14, The pre-commit hook definition
for entry "uv lock --locked" currently uses a root-anchored files pattern and
pass_filenames: false, so it only validates the repo root; change the hook so it
runs per changed file and matches nested projects by (1) setting pass_filenames:
true and (2) replacing the files regex that references
'^uv\.lock$|^pyproject\.toml$|^uv\.toml$' with a non-root-anchored pattern that
matches nested paths (e.g. allow files ending with uv.lock, pyproject.toml or
uv.toml anywhere in the tree), keeping the entry "uv lock --locked" unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant